<!DOCTYPE html>
<!-- saved from url=(0076)https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html -->
<html class=" js flexbox canvas canvastext webgl no-touch geolocation postmessage websqldatabase indexeddb hashchange history draganddrop websockets rgba hsla multiplebgs backgroundsize borderimage borderradius boxshadow textshadow opacity cssanimations csscolumns cssgradients cssreflections csstransforms csstransforms3d csstransitions fontface generatedcontent video audio localstorage sessionstorage webworkers applicationcache no-svg no-inlinesvg no-smil no-svgclippaths" lang="en" style=""><!--<![endif]--><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.1.0 documentation</title>
<style class="anchorjs"></style><link href="./seq2seq_files/theme.css" rel="stylesheet" type="text/css">
<!-- <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> -->
<link href="./seq2seq_files/gallery.css" rel="stylesheet" type="text/css">
<link href="https://pytorch.org/tutorials/genindex.html" rel="index" title="Index">
<link href="https://pytorch.org/tutorials/search.html" rel="search" title="Search">
<link href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="next" title="DCGAN Tutorial">
<link href="https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html" rel="prev" title="Advanced: Making Dynamic Decisions and the Bi-LSTM CRF">
<script type="text/javascript" async="" src="./seq2seq_files/analytics.js"></script><script src="./seq2seq_files/inferredEvents.js" async=""></script><script src="./seq2seq_files/243028289693773" async=""></script><script async="" src="./seq2seq_files/fbevents.js"></script><script async="" src="./seq2seq_files/analytics.js"></script><script src="./seq2seq_files/modernizr.min.js"></script>
<style type="text/css">.MathJax_Hover_Frame {border-radius: .25em; -webkit-border-radius: .25em; -moz-border-radius: .25em; -khtml-border-radius: .25em; box-shadow: 0px 0px 15px #83A; -webkit-box-shadow: 0px 0px 15px #83A; -moz-box-shadow: 0px 0px 15px #83A; -khtml-box-shadow: 0px 0px 15px #83A; border: 1px solid #A6D ! important; display: inline-block; position: absolute}
.MathJax_Menu_Button .MathJax_Hover_Arrow {position: absolute; cursor: pointer; display: inline-block; border: 2px solid #AAA; border-radius: 4px; -webkit-border-radius: 4px; -moz-border-radius: 4px; -khtml-border-radius: 4px; font-family: 'Courier New',Courier; font-size: 9px; color: #F0F0F0}
.MathJax_Menu_Button .MathJax_Hover_Arrow span {display: block; background-color: #AAA; border: 1px solid; border-radius: 3px; line-height: 0; padding: 4px}
.MathJax_Hover_Arrow:hover {color: white!important; border: 2px solid #CCC!important}
.MathJax_Hover_Arrow:hover span {background-color: #CCC!important}
</style><style type="text/css">#MathJax_About {position: fixed; left: 50%; width: auto; text-align: center; border: 3px outset; padding: 1em 2em; background-color: #DDDDDD; color: black; cursor: default; font-family: message-box; font-size: 120%; font-style: normal; text-indent: 0; text-transform: none; line-height: normal; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; z-index: 201; border-radius: 15px; -webkit-border-radius: 15px; -moz-border-radius: 15px; -khtml-border-radius: 15px; box-shadow: 0px 10px 20px #808080; -webkit-box-shadow: 0px 10px 20px #808080; -moz-box-shadow: 0px 10px 20px #808080; -khtml-box-shadow: 0px 10px 20px #808080; filter: progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color='gray', Positive='true')}
#MathJax_About.MathJax_MousePost {outline: none}
.MathJax_Menu {position: absolute; background-color: white; color: black; width: auto; padding: 5px 0px; border: 1px solid #CCCCCC; margin: 0; cursor: default; font: menu; text-align: left; text-indent: 0; text-transform: none; line-height: normal; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; z-index: 201; border-radius: 5px; -webkit-border-radius: 5px; -moz-border-radius: 5px; -khtml-border-radius: 5px; box-shadow: 0px 10px 20px #808080; -webkit-box-shadow: 0px 10px 20px #808080; -moz-box-shadow: 0px 10px 20px #808080; -khtml-box-shadow: 0px 10px 20px #808080; filter: progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color='gray', Positive='true')}
.MathJax_MenuItem {padding: 1px 2em; background: transparent}
.MathJax_MenuArrow {position: absolute; right: .5em; padding-top: .25em; color: #666666; font-size: .75em}
.MathJax_MenuActive .MathJax_MenuArrow {color: white}
.MathJax_MenuArrow.RTL {left: .5em; right: auto}
.MathJax_MenuCheck {position: absolute; left: .7em}
.MathJax_MenuCheck.RTL {right: .7em; left: auto}
.MathJax_MenuRadioCheck {position: absolute; left: .7em}
.MathJax_MenuRadioCheck.RTL {right: .7em; left: auto}
.MathJax_MenuLabel {padding: 1px 2em 3px 1.33em; font-style: italic}
.MathJax_MenuRule {border-top: 1px solid #DDDDDD; margin: 4px 3px}
.MathJax_MenuDisabled {color: GrayText}
.MathJax_MenuActive {background-color: #606872; color: white}
.MathJax_MenuDisabled:focus, .MathJax_MenuLabel:focus {background-color: #E8E8E8}
.MathJax_ContextMenu:focus {outline: none}
.MathJax_ContextMenu .MathJax_MenuItem:focus {outline: none}
#MathJax_AboutClose {top: .2em; right: .2em}
.MathJax_Menu .MathJax_MenuClose {top: -10px; left: -10px}
.MathJax_MenuClose {position: absolute; cursor: pointer; display: inline-block; border: 2px solid #AAA; border-radius: 18px; -webkit-border-radius: 18px; -moz-border-radius: 18px; -khtml-border-radius: 18px; font-family: 'Courier New',Courier; font-size: 24px; color: #F0F0F0}
.MathJax_MenuClose span {display: block; background-color: #AAA; border: 1.5px solid; border-radius: 18px; -webkit-border-radius: 18px; -moz-border-radius: 18px; -khtml-border-radius: 18px; line-height: 0; padding: 8px 0 6px}
.MathJax_MenuClose:hover {color: white!important; border: 2px solid #CCC!important}
.MathJax_MenuClose:hover span {background-color: #CCC!important}
.MathJax_MenuClose:hover:focus {outline: none}
</style><style type="text/css">.MathJax_Preview .MJXf-math {color: inherit!important}
</style><style type="text/css">.MJX_Assistive_MathML {position: absolute!important; top: 0; left: 0; clip: rect(1px, 1px, 1px, 1px); padding: 1px 0 0 0!important; border: 0!important; height: 1px!important; width: 1px!important; overflow: hidden!important; display: block!important; -webkit-touch-callout: none; -webkit-user-select: none; -khtml-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none}
.MJX_Assistive_MathML.MJX_Assistive_MathML_Block {width: 100%!important}
</style><style type="text/css">#MathJax_Zoom {position: absolute; background-color: #F0F0F0; overflow: auto; display: block; z-index: 301; padding: .5em; border: 1px solid black; margin: 0; font-weight: normal; font-style: normal; text-align: left; text-indent: 0; text-transform: none; line-height: normal; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; -webkit-box-sizing: content-box; -moz-box-sizing: content-box; box-sizing: content-box; box-shadow: 5px 5px 15px #AAAAAA; -webkit-box-shadow: 5px 5px 15px #AAAAAA; -moz-box-shadow: 5px 5px 15px #AAAAAA; -khtml-box-shadow: 5px 5px 15px #AAAAAA; filter: progid:DXImageTransform.Microsoft.dropshadow(OffX=2, OffY=2, Color='gray', Positive='true')}
#MathJax_ZoomOverlay {position: absolute; left: 0; top: 0; z-index: 300; display: inline-block; width: 100%; height: 100%; border: 0; padding: 0; margin: 0; background-color: white; opacity: 0; filter: alpha(opacity=0)}
#MathJax_ZoomFrame {position: relative; display: inline-block; height: 0; width: 0}
#MathJax_ZoomEventTrap {position: absolute; left: 0; top: 0; z-index: 302; display: inline-block; border: 0; padding: 0; margin: 0; background-color: white; opacity: 0; filter: alpha(opacity=0)}
</style><style type="text/css">.MathJax_Preview {color: #888}
#MathJax_Message {position: fixed; left: 1em; bottom: 1.5em; background-color: #E6E6E6; border: 1px solid #959595; margin: 0px; padding: 2px 8px; z-index: 102; color: black; font-size: 80%; width: auto; white-space: nowrap}
#MathJax_MSIE_Frame {position: absolute; top: 0; left: 0; width: 0px; z-index: 101; border: 0px; margin: 0px; padding: 0px}
.MathJax_Error {color: #CC0000; font-style: italic}
</style><style type="text/css">.MJXp-script {font-size: .8em}
.MJXp-right {-webkit-transform-origin: right; -moz-transform-origin: right; -ms-transform-origin: right; -o-transform-origin: right; transform-origin: right}
.MJXp-bold {font-weight: bold}
.MJXp-italic {font-style: italic}
.MJXp-scr {font-family: MathJax_Script,'Times New Roman',Times,STIXGeneral,serif}
.MJXp-frak {font-family: MathJax_Fraktur,'Times New Roman',Times,STIXGeneral,serif}
.MJXp-sf {font-family: MathJax_SansSerif,'Times New Roman',Times,STIXGeneral,serif}
.MJXp-cal {font-family: MathJax_Caligraphic,'Times New Roman',Times,STIXGeneral,serif}
.MJXp-mono {font-family: MathJax_Typewriter,'Times New Roman',Times,STIXGeneral,serif}
.MJXp-largeop {font-size: 150%}
.MJXp-largeop.MJXp-int {vertical-align: -.2em}
.MJXp-math {display: inline-block; line-height: 1.2; text-indent: 0; font-family: 'Times New Roman',Times,STIXGeneral,serif; white-space: nowrap; border-collapse: collapse}
.MJXp-display {display: block; text-align: center; margin: 1em 0}
.MJXp-math span {display: inline-block}
.MJXp-box {display: block!important; text-align: center}
.MJXp-box:after {content: " "}
.MJXp-rule {display: block!important; margin-top: .1em}
.MJXp-char {display: block!important}
.MJXp-mo {margin: 0 .15em}
.MJXp-mfrac {margin: 0 .125em; vertical-align: .25em}
.MJXp-denom {display: inline-table!important; width: 100%}
.MJXp-denom > * {display: table-row!important}
.MJXp-surd {vertical-align: top}
.MJXp-surd > * {display: block!important}
.MJXp-script-box > *  {display: table!important; height: 50%}
.MJXp-script-box > * > * {display: table-cell!important; vertical-align: top}
.MJXp-script-box > *:last-child > * {vertical-align: bottom}
.MJXp-script-box > * > * > * {display: block!important}
.MJXp-mphantom {visibility: hidden}
.MJXp-munderover {display: inline-table!important}
.MJXp-over {display: inline-block!important; text-align: center}
.MJXp-over > * {display: block!important}
.MJXp-munderover > * {display: table-row!important}
.MJXp-mtable {vertical-align: .25em; margin: 0 .125em}
.MJXp-mtable > * {display: inline-table!important; vertical-align: middle}
.MJXp-mtr {display: table-row!important}
.MJXp-mtd {display: table-cell!important; text-align: center; padding: .5em 0 0 .5em}
.MJXp-mtr > .MJXp-mtd:first-child {padding-left: 0}
.MJXp-mtr:first-child > .MJXp-mtd {padding-top: 0}
.MJXp-mlabeledtr {display: table-row!important}
.MJXp-mlabeledtr > .MJXp-mtd:first-child {padding-left: 0}
.MJXp-mlabeledtr:first-child > .MJXp-mtd {padding-top: 0}
.MJXp-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 1px 3px; font-style: normal; font-size: 90%}
.MJXp-scale0 {-webkit-transform: scaleX(.0); -moz-transform: scaleX(.0); -ms-transform: scaleX(.0); -o-transform: scaleX(.0); transform: scaleX(.0)}
.MJXp-scale1 {-webkit-transform: scaleX(.1); -moz-transform: scaleX(.1); -ms-transform: scaleX(.1); -o-transform: scaleX(.1); transform: scaleX(.1)}
.MJXp-scale2 {-webkit-transform: scaleX(.2); -moz-transform: scaleX(.2); -ms-transform: scaleX(.2); -o-transform: scaleX(.2); transform: scaleX(.2)}
.MJXp-scale3 {-webkit-transform: scaleX(.3); -moz-transform: scaleX(.3); -ms-transform: scaleX(.3); -o-transform: scaleX(.3); transform: scaleX(.3)}
.MJXp-scale4 {-webkit-transform: scaleX(.4); -moz-transform: scaleX(.4); -ms-transform: scaleX(.4); -o-transform: scaleX(.4); transform: scaleX(.4)}
.MJXp-scale5 {-webkit-transform: scaleX(.5); -moz-transform: scaleX(.5); -ms-transform: scaleX(.5); -o-transform: scaleX(.5); transform: scaleX(.5)}
.MJXp-scale6 {-webkit-transform: scaleX(.6); -moz-transform: scaleX(.6); -ms-transform: scaleX(.6); -o-transform: scaleX(.6); transform: scaleX(.6)}
.MJXp-scale7 {-webkit-transform: scaleX(.7); -moz-transform: scaleX(.7); -ms-transform: scaleX(.7); -o-transform: scaleX(.7); transform: scaleX(.7)}
.MJXp-scale8 {-webkit-transform: scaleX(.8); -moz-transform: scaleX(.8); -ms-transform: scaleX(.8); -o-transform: scaleX(.8); transform: scaleX(.8)}
.MJXp-scale9 {-webkit-transform: scaleX(.9); -moz-transform: scaleX(.9); -ms-transform: scaleX(.9); -o-transform: scaleX(.9); transform: scaleX(.9)}
.MathJax_PHTML .noError {vertical-align: ; font-size: 90%; text-align: left; color: black; padding: 1px 3px; border: 1px solid}
</style></head>
<body class="pytorch-body"><div id="MathJax_Message" style="display: none;"></div><div class="container-fluid header-holder tutorials-header" id="header-holder">
<div class="container">
<div class="header-container">
<a aria-label="PyTorch" class="header-logo" href="https://pytorch.org/"></a>
<div class="main-menu">
<ul>
<li>
<a href="https://pytorch.org/get-started">Get Started</a>
</li>
<li>
<a href="https://pytorch.org/features">Features</a>
</li>
<li>
<a href="https://pytorch.org/ecosystem">Ecosystem</a>
</li>
<li>
<a href="https://pytorch.org/blog/">Blog</a>
</li>
<li class="active">
<a href="https://pytorch.org/tutorials">Tutorials</a>
</li>
<li>
<a href="https://pytorch.org/docs/stable/index.html">Docs</a>
</li>
<li>
<a href="https://pytorch.org/resources">Resources</a>
</li>
<li>
<a href="https://github.com/pytorch/pytorch">Github</a>
</li>
</ul>
</div>
<a class="main-menu-open-button" data-behavior="open-mobile-menu" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#"></a>
</div>
</div>
</div>

<div class="table-of-contents-link-wrapper">
<span>Table of Contents</span>
<a class="toggle-table-of-contents" data-behavior="toggle-table-of-contents" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#"></a>
</div>
<nav class="pytorch-left-menu make-fixed" data-toggle="wy-nav-shift" id="pytorch-left-menu" style="height: 100%;">
<div class="pytorch-side-scroll">
<div aria-label="main navigation" class="pytorch-menu pytorch-menu-vertical" data-spy="affix" role="navigation">
<div class="pytorch-left-menu-search">
<div class="version">
                  1.1.0
                </div>
<div role="search">
<form action="https://pytorch.org/tutorials/search.html" class="wy-form" id="rtd-search-form" method="get">
<input name="q" placeholder="Search Tutorials" type="text">
<input name="check_keywords" type="hidden" value="yes">
<input name="area" type="hidden" value="default">
</form>
</div>
</div>
<p class="caption"><span class="caption-text">Getting Started</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html">Deep Learning with PyTorch: A 60 Minute Blitz</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">Data Loading and Processing Tutorial</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html">Learning PyTorch with Examples</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html">Transfer Learning Tutorial</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/deploy_seq2seq_hybrid_frontend_tutorial.html">Deploying a Seq2Seq Model with the Hybrid Frontend</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">Saving and Loading Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/nn_tutorial.html">What is <cite>torch.nn</cite> <em>really</em>?</a></li>
</ul>
<p class="caption"><span class="caption-text">Image</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">TorchVision 0.3 Object Detection Finetuning Tutorial</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">Finetuning Torchvision Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html">Spatial Transformer Networks Tutorial</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html">Neural Transfer Using PyTorch</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/fgsm_tutorial.html">Adversarial Example Generation</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html">Transfering a Model from PyTorch to Caffe2 and Mobile using ONNX</a></li>
</ul>
<p class="caption"><span class="caption-text">Text</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html">Chatbot Tutorial</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html">Generating Names with a Character-Level RNN</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html">Classifying Names with a Character-Level RNN</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/deep_learning_nlp_tutorial.html">Deep Learning for NLP with Pytorch</a></li>
<li class="toctree-l1 current"><a class="reference internal current" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#">Translation with a Sequence to Sequence Network and Attention</a></li>
</ul>
<p class="caption"><span class="caption-text">Generative</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html">DCGAN Tutorial</a></li>
</ul>
<p class="caption"><span class="caption-text">Reinforcement Learning</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html">Reinforcement Learning (DQN) Tutorial</a></li>
</ul>
<p class="caption"><span class="caption-text">Extending PyTorch</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html">Creating Extensions Using numpy and scipy</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/cpp_extension.html">Custom C++ and CUDA Extensions</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html">Extending TorchScript with Custom C++ Operators</a></li>
</ul>
<p class="caption"><span class="caption-text">Production Usage</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html">Model Parallel Best Practices</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">Getting Started with Distributed Data Parallel</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/dist_tuto.html">Writing Distributed Applications with PyTorch</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html">PyTorch 1.0 Distributed Trainer with Amazon AWS</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/ONNXLive.html">ONNX Live Tutorial</a></li>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/cpp_export.html">Loading a PyTorch Model in C++</a></li>
</ul>
<p class="caption"><span class="caption-text">PyTorch in Other Languages</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="https://pytorch.org/tutorials/advanced/cpp_frontend.html">Using the PyTorch C++ Frontend</a></li>
</ul>
</div>
</div>
</nav>
<div class="pytorch-container">
<div class="pytorch-page-level-bar left-menu-is-fixed" id="pytorch-page-level-bar">
<div class="pytorch-breadcrumbs-wrapper">
<div aria-label="breadcrumbs navigation" role="navigation">
<ul class="pytorch-breadcrumbs">
<li>
<a href="https://pytorch.org/tutorials/index.html">
          
            Tutorials
          
        </a> &gt;
      </li>
<li>Translation with a Sequence to Sequence Network and Attention</li>
<li class="pytorch-breadcrumbs-aside">
<a href="https://pytorch.org/tutorials/_sources/intermediate/seq2seq_translation_tutorial.rst.txt" rel="nofollow"><img src="./seq2seq_files/view-page-source-icon.svg"></a>
</li>
</ul>
</div>
</div>
<div class="pytorch-shortcuts-wrapper" id="pytorch-shortcuts-wrapper" style="display: block;">
          Shortcuts
        </div>
</div>
<section class="pytorch-content-wrap" data-toggle="wy-nav-shift" id="pytorch-content-wrap">
<div class="pytorch-content-left">
<div class="pytorch-call-to-action-links">
<div id="tutorial-type">intermediate/seq2seq_translation_tutorial</div>
<a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/seq2seq_translation_tutorial.ipynb" data-behavior="call-to-action-event" data-response="Run in Google Colab" target="_blank"><div id="google-colab-link">
<img class="call-to-action-img" src="./seq2seq_files/pytorch-colab.svg">
<div class="call-to-action-desktop-view">Run in Google Colab</div>
<div class="call-to-action-mobile-view">Colab</div>
</div></a>
<a href="https://pytorch.org/tutorials/_downloads/a60617788061539b5449701ae76aee56/seq2seq_translation_tutorial.ipynb" data-behavior="call-to-action-event" data-response="Download Notebook"><div id="download-notebook-link">
<img class="call-to-action-notebook-img" src="./seq2seq_files/pytorch-download.svg">
<div class="call-to-action-desktop-view">Download Notebook</div>
<div class="call-to-action-mobile-view">Notebook</div>
</div></a>
<a href="https://github.com/pytorch/tutorials/blob/master/intermediate_source/seq2seq_translation_tutorial.py" data-behavior="call-to-action-event" data-response="View on Github" target="_blank"><div id="github-view-link">
<img class="call-to-action-img" src="./seq2seq_files/pytorch-github.svg">
<div class="call-to-action-desktop-view">View on GitHub</div>
<div class="call-to-action-mobile-view">GitHub</div>
</div></a>
</div>
<div class="rst-content">
<div class="main-content" itemscope="itemscope" itemtype="http://schema.org/Article" role="main">
<article class="pytorch-article" id="pytorch-article" itemprop="articleBody">
<div class="sphx-glr-download-link-note admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Click <a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-download-intermediate-seq2seq-translation-tutorial-py"><span class="std std-ref">here</span></a> to download the full example code</p>
</div>
<div class="sphx-glr-example-title section" id="translation-with-a-sequence-to-sequence-network-and-attention">
<span id="sphx-glr-intermediate-seq2seq-translation-tutorial-py"></span><h1>Translation with a Sequence to Sequence Network and Attention<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#translation-with-a-sequence-to-sequence-network-and-attention" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h1>
<p><strong>Author</strong>: <a class="reference external" href="https://github.com/spro/practical-pytorch">Sean Robertson</a></p>
<p>In this project we will be teaching a neural network to translate from
French to English.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>[KEY: &gt; input, = target, &lt; output]

&gt; il est en train de peindre un tableau .
= he is painting a picture .
&lt; he is painting a picture .

&gt; pourquoi ne pas essayer ce vin delicieux ?
= why not try that delicious wine ?
&lt; why not try that delicious wine ?

&gt; elle n est pas poete mais romanciere .
= she is not a poet but a novelist .
&lt; she not not a poet but a novelist .

&gt; vous etes trop maigre .
= you re too skinny .
&lt; you re all alone .
</pre></div>
</div>
<p>… to varying degrees of success.</p>
<p>This is made possible by the simple but powerful idea of the <a class="reference external" href="https://arxiv.org/abs/1409.3215">sequence
to sequence network</a>, in which two
recurrent neural networks work together to transform one sequence to
another. An encoder network condenses an input sequence into a vector,
and a decoder network unfolds that vector into a new sequence.</p>
<div class="figure">
<img alt="" src="./seq2seq_files/seq2seq.png">
</div>
<p>To improve upon this model we’ll use an <a class="reference external" href="https://arxiv.org/abs/1409.0473">attention
mechanism</a>, which lets the decoder
learn to focus over a specific range of the input sequence.</p>
<p><strong>Recommended Reading:</strong></p>
<p>I assume you have at least installed PyTorch, know Python, and
understand Tensors:</p>
<ul class="simple">
<li><a class="reference external" href="https://pytorch.org/">https://pytorch.org/</a> For installation instructions</li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html"><span class="doc">Deep Learning with PyTorch: A 60 Minute Blitz</span></a> to get started with PyTorch in general</li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html"><span class="doc">Learning PyTorch with Examples</span></a> for a wide and deep overview</li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/beginner/former_torchies_tutorial.html"><span class="doc">PyTorch for Former Torch Users</span></a> if you are former Lua Torch user</li>
</ul>
<p>It would also be useful to know about Sequence to Sequence networks and
how they work:</p>
<ul class="simple">
<li><a class="reference external" href="https://arxiv.org/abs/1406.1078">Learning Phrase Representations using RNN Encoder-Decoder for
Statistical Machine Translation</a></li>
<li><a class="reference external" href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural
Networks</a></li>
<li><a class="reference external" href="https://arxiv.org/abs/1409.0473">Neural Machine Translation by Jointly Learning to Align and
Translate</a></li>
<li><a class="reference external" href="https://arxiv.org/abs/1506.05869">A Neural Conversational Model</a></li>
</ul>
<p>You will also find the previous tutorials on
<a class="reference internal" href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html"><span class="doc">Classifying Names with a Character-Level RNN</span></a>
and <a class="reference internal" href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html"><span class="doc">Generating Names with a Character-Level RNN</span></a>
helpful as those concepts are very similar to the Encoder and Decoder
models, respectively.</p>
<p>And for more, read the papers that introduced these topics:</p>
<ul class="simple">
<li><a class="reference external" href="https://arxiv.org/abs/1406.1078">Learning Phrase Representations using RNN Encoder-Decoder for
Statistical Machine Translation</a></li>
<li><a class="reference external" href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural
Networks</a></li>
<li><a class="reference external" href="https://arxiv.org/abs/1409.0473">Neural Machine Translation by Jointly Learning to Align and
Translate</a></li>
<li><a class="reference external" href="https://arxiv.org/abs/1506.05869">A Neural Conversational Model</a></li>
</ul>
<p><strong>Requirements</strong></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">unicode_literals</span><span class="p">,</span> <span class="n">print_function</span><span class="p">,</span> <span class="n">division</span>
<span class="kn">from</span> <span class="nn">io</span> <span class="k">import</span> <span class="nb">open</span>
<span class="kn">import</span> <span class="nn">unicodedata</span>
<span class="kn">import</span> <span class="nn">string</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">random</span>

<span class="kn">import</span> <span class="nn">torch</span>
<span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
<span class="kn">from</span> <span class="nn">torch</span> <span class="k">import</span> <span class="n">optim</span>
<span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">F</span>

<span class="n">device</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="s2">"cuda"</span> <span class="k">if</span> <span class="n">torch</span><span class="o">.</span><span class="n">cuda</span><span class="o">.</span><span class="n">is_available</span><span class="p">()</span> <span class="k">else</span> <span class="s2">"cpu"</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="loading-data-files">
<h2>Loading data files<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#loading-data-files" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h2>
<p>The data for this project is a set of many thousands of English to
French translation pairs.</p>
<p><a class="reference external" href="https://opendata.stackexchange.com/questions/3888/dataset-of-sentences-translated-into-many-languages">This question on Open Data Stack
Exchange</a>
pointed me to the open translation site <a class="reference external" href="https://tatoeba.org/">https://tatoeba.org/</a> which has
downloads available at <a class="reference external" href="https://tatoeba.org/eng/downloads">https://tatoeba.org/eng/downloads</a> - and better
yet, someone did the extra work of splitting language pairs into
individual text files here: <a class="reference external" href="https://www.manythings.org/anki/">https://www.manythings.org/anki/</a></p>
<p>The English to French pairs are too big to include in the repo, so
download to <code class="docutils literal notranslate"><span class="pre">data/eng-fra.txt</span></code> before continuing. The file is a tab
separated list of translation pairs:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">I</span> <span class="n">am</span> <span class="n">cold</span><span class="o">.</span>    <span class="n">J</span><span class="s1">'ai froid.</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Download the data from
<a class="reference external" href="https://download.pytorch.org/tutorial/data.zip">here</a>
and extract it to the current directory.</p>
</div>
<p>Similar to the character encoding used in the character-level RNN
tutorials, we will be representing each word in a language as a one-hot
vector, or giant vector of zeros except for a single one (at the index
of the word). Compared to the dozens of characters that might exist in a
language, there are many many more words, so the encoding vector is much
larger. We will however cheat a bit and trim the data to only use a few
thousand words per language.</p>
<div class="figure">
<img alt="" src="./seq2seq_files/word-encoding.png">
</div>
<p>We’ll need a unique index per word to use as the inputs and targets of
the networks later. To keep track of all this we will use a helper class
called <code class="docutils literal notranslate"><span class="pre">Lang</span></code> which has word → index (<code class="docutils literal notranslate"><span class="pre">word2index</span></code>) and index → word
(<code class="docutils literal notranslate"><span class="pre">index2word</span></code>) dictionaries, as well as a count of each word
<code class="docutils literal notranslate"><span class="pre">word2count</span></code> to use to later replace rare words.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">SOS_token</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">EOS_token</span> <span class="o">=</span> <span class="mi">1</span>


<span class="k">class</span> <span class="nc">Lang</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">word2index</span> <span class="o">=</span> <span class="p">{}</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">word2count</span> <span class="o">=</span> <span class="p">{}</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">index2word</span> <span class="o">=</span> <span class="p">{</span><span class="mi">0</span><span class="p">:</span> <span class="s2">"SOS"</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="s2">"EOS"</span><span class="p">}</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_words</span> <span class="o">=</span> <span class="mi">2</span>  <span class="c1"># Count SOS and EOS</span>

    <span class="k">def</span> <span class="nf">addSentence</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sentence</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">sentence</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">):</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">addWord</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">addWord</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">word</span><span class="p">):</span>
        <span class="k">if</span> <span class="n">word</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">word2index</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">word2index</span><span class="p">[</span><span class="n">word</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_words</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">word2count</span><span class="p">[</span><span class="n">word</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">index2word</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">n_words</span><span class="p">]</span> <span class="o">=</span> <span class="n">word</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">n_words</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">word2count</span><span class="p">[</span><span class="n">word</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>
</div>
<p>The files are all in Unicode, to simplify we will turn Unicode
characters to ASCII, make everything lowercase, and trim most
punctuation.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># Turn a Unicode string to plain ASCII, thanks to</span>
<span class="c1"># https://stackoverflow.com/a/518232/2809427</span>
<span class="k">def</span> <span class="nf">unicodeToAscii</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
    <span class="k">return</span> <span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
        <span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">unicodedata</span><span class="o">.</span><span class="n">normalize</span><span class="p">(</span><span class="s1">'NFD'</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">unicodedata</span><span class="o">.</span><span class="n">category</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">!=</span> <span class="s1">'Mn'</span>
    <span class="p">)</span>

<span class="c1"># Lowercase, trim, and remove non-letter characters</span>


<span class="k">def</span> <span class="nf">normalizeString</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">unicodeToAscii</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">"([.!?])"</span><span class="p">,</span> <span class="sa">r</span><span class="s2">" \1"</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">"[^a-zA-Z.!?]+"</span><span class="p">,</span> <span class="sa">r</span><span class="s2">" "</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">s</span>
</pre></div>
</div>
<p>To read the data file we will split the file into lines, and then split
lines into pairs. The files are all English → Other Language, so if we
want to translate from Other Language → English I added the <code class="docutils literal notranslate"><span class="pre">reverse</span></code>
flag to reverse the pairs.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">readLangs</span><span class="p">(</span><span class="n">lang1</span><span class="p">,</span> <span class="n">lang2</span><span class="p">,</span> <span class="n">reverse</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"Reading lines..."</span><span class="p">)</span>

    <span class="c1"># Read the file and split into lines</span>
    <span class="n">lines</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'data/</span><span class="si">%s</span><span class="s1">-</span><span class="si">%s</span><span class="s1">.txt'</span> <span class="o">%</span> <span class="p">(</span><span class="n">lang1</span><span class="p">,</span> <span class="n">lang2</span><span class="p">),</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span>\
        <span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>

    <span class="c1"># Split every line into pairs and normalize</span>
    <span class="n">pairs</span> <span class="o">=</span> <span class="p">[[</span><span class="n">normalizeString</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">l</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">)]</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">]</span>

    <span class="c1"># Reverse pairs, make Lang instances</span>
    <span class="k">if</span> <span class="n">reverse</span><span class="p">:</span>
        <span class="n">pairs</span> <span class="o">=</span> <span class="p">[</span><span class="nb">list</span><span class="p">(</span><span class="nb">reversed</span><span class="p">(</span><span class="n">p</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">pairs</span><span class="p">]</span>
        <span class="n">input_lang</span> <span class="o">=</span> <span class="n">Lang</span><span class="p">(</span><span class="n">lang2</span><span class="p">)</span>
        <span class="n">output_lang</span> <span class="o">=</span> <span class="n">Lang</span><span class="p">(</span><span class="n">lang1</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">input_lang</span> <span class="o">=</span> <span class="n">Lang</span><span class="p">(</span><span class="n">lang1</span><span class="p">)</span>
        <span class="n">output_lang</span> <span class="o">=</span> <span class="n">Lang</span><span class="p">(</span><span class="n">lang2</span><span class="p">)</span>

    <span class="k">return</span> <span class="n">input_lang</span><span class="p">,</span> <span class="n">output_lang</span><span class="p">,</span> <span class="n">pairs</span>
</pre></div>
</div>
<p>Since there are a <em>lot</em> of example sentences and we want to train
something quickly, we’ll trim the data set to only relatively short and
simple sentences. Here the maximum length is 10 words (that includes
ending punctuation) and we’re filtering to sentences that translate to
the form “I am” or “He is” etc. (accounting for apostrophes replaced
earlier).</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">MAX_LENGTH</span> <span class="o">=</span> <span class="mi">10</span>

<span class="n">eng_prefixes</span> <span class="o">=</span> <span class="p">(</span>
    <span class="s2">"i am "</span><span class="p">,</span> <span class="s2">"i m "</span><span class="p">,</span>
    <span class="s2">"he is"</span><span class="p">,</span> <span class="s2">"he s "</span><span class="p">,</span>
    <span class="s2">"she is"</span><span class="p">,</span> <span class="s2">"she s "</span><span class="p">,</span>
    <span class="s2">"you are"</span><span class="p">,</span> <span class="s2">"you re "</span><span class="p">,</span>
    <span class="s2">"we are"</span><span class="p">,</span> <span class="s2">"we re "</span><span class="p">,</span>
    <span class="s2">"they are"</span><span class="p">,</span> <span class="s2">"they re "</span>
<span class="p">)</span>


<span class="k">def</span> <span class="nf">filterPair</span><span class="p">(</span><span class="n">p</span><span class="p">):</span>
    <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">p</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">))</span> <span class="o">&lt;</span> <span class="n">MAX_LENGTH</span> <span class="ow">and</span> \
        <span class="nb">len</span><span class="p">(</span><span class="n">p</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">))</span> <span class="o">&lt;</span> <span class="n">MAX_LENGTH</span> <span class="ow">and</span> \
        <span class="n">p</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">eng_prefixes</span><span class="p">)</span>


<span class="k">def</span> <span class="nf">filterPairs</span><span class="p">(</span><span class="n">pairs</span><span class="p">):</span>
    <span class="k">return</span> <span class="p">[</span><span class="n">pair</span> <span class="k">for</span> <span class="n">pair</span> <span class="ow">in</span> <span class="n">pairs</span> <span class="k">if</span> <span class="n">filterPair</span><span class="p">(</span><span class="n">pair</span><span class="p">)]</span>
</pre></div>
</div>
<p>The full process for preparing the data is:</p>
<ul class="simple">
<li>Read text file and split into lines, split lines into pairs</li>
<li>Normalize text, filter by length and content</li>
<li>Make word lists from sentences in pairs</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">prepareData</span><span class="p">(</span><span class="n">lang1</span><span class="p">,</span> <span class="n">lang2</span><span class="p">,</span> <span class="n">reverse</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
    <span class="n">input_lang</span><span class="p">,</span> <span class="n">output_lang</span><span class="p">,</span> <span class="n">pairs</span> <span class="o">=</span> <span class="n">readLangs</span><span class="p">(</span><span class="n">lang1</span><span class="p">,</span> <span class="n">lang2</span><span class="p">,</span> <span class="n">reverse</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"Read </span><span class="si">%s</span><span class="s2"> sentence pairs"</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">pairs</span><span class="p">))</span>
    <span class="n">pairs</span> <span class="o">=</span> <span class="n">filterPairs</span><span class="p">(</span><span class="n">pairs</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"Trimmed to </span><span class="si">%s</span><span class="s2"> sentence pairs"</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">pairs</span><span class="p">))</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"Counting words..."</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">pair</span> <span class="ow">in</span> <span class="n">pairs</span><span class="p">:</span>
        <span class="n">input_lang</span><span class="o">.</span><span class="n">addSentence</span><span class="p">(</span><span class="n">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
        <span class="n">output_lang</span><span class="o">.</span><span class="n">addSentence</span><span class="p">(</span><span class="n">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">"Counted words:"</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">input_lang</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">input_lang</span><span class="o">.</span><span class="n">n_words</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">output_lang</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">output_lang</span><span class="o">.</span><span class="n">n_words</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">input_lang</span><span class="p">,</span> <span class="n">output_lang</span><span class="p">,</span> <span class="n">pairs</span>


<span class="n">input_lang</span><span class="p">,</span> <span class="n">output_lang</span><span class="p">,</span> <span class="n">pairs</span> <span class="o">=</span> <span class="n">prepareData</span><span class="p">(</span><span class="s1">'eng'</span><span class="p">,</span> <span class="s1">'fra'</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">pairs</span><span class="p">))</span>
</pre></div>
</div>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>Reading lines...
Read 135842 sentence pairs
Trimmed to 10599 sentence pairs
Counting words...
Counted words:
fra 4345
eng 2803
['tu n es pas le bienvenu ici .', 'you re not welcome here .']
</pre></div>
</div>
</div>
<div class="section" id="the-seq2seq-model">
<h2>The Seq2Seq Model<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-seq2seq-model" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h2>
<p>A Recurrent Neural Network, or RNN, is a network that operates on a
sequence and uses its own output as input for subsequent steps.</p>
<p>A <a class="reference external" href="https://arxiv.org/abs/1409.3215">Sequence to Sequence network</a>, or
seq2seq network, or <a class="reference external" href="https://arxiv.org/pdf/1406.1078v3.pdf">Encoder Decoder
network</a>, is a model
consisting of two RNNs called the encoder and decoder. The encoder reads
an input sequence and outputs a single vector, and the decoder reads
that vector to produce an output sequence.</p>
<div class="figure">
<img alt="" src="./seq2seq_files/seq2seq.png">
</div>
<p>Unlike sequence prediction with a single RNN, where every input
corresponds to an output, the seq2seq model frees us from sequence
length and order, which makes it ideal for translation between two
languages.</p>
<p>Consider the sentence “Je ne suis pas le chat noir” → “I am not the
black cat”. Most of the words in the input sentence have a direct
translation in the output sentence, but are in slightly different
orders, e.g. “chat noir” and “black cat”. Because of the “ne/pas”
construction there is also one more word in the input sentence. It would
be difficult to produce a correct translation directly from the sequence
of input words.</p>
<p>With a seq2seq model the encoder creates a single vector which, in the
ideal case, encodes the “meaning” of the input sequence into a single
vector — a single point in some N dimensional space of sentences.</p>
<div class="section" id="the-encoder">
<h3>The Encoder<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-encoder" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h3>
<p>The encoder of a seq2seq network is a RNN that outputs some value for
every word from the input sentence. For every input word the encoder
outputs a vector and a hidden state, and uses the hidden state for the
next input word.</p>
<div class="figure">
<img alt="" src="./seq2seq_files/encoder-network.png">
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">EncoderRNN</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_size</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">):</span>
        <span class="nb">super</span><span class="p">(</span><span class="n">EncoderRNN</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span> <span class="o">=</span> <span class="n">hidden_size</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">embedding</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="n">input_size</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">gru</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">GRU</span><span class="p">(</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">input</span><span class="p">,</span> <span class="n">hidden</span><span class="p">):</span>
        <span class="n">embedded</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">embedding</span><span class="p">(</span><span class="nb">input</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
        <span class="n">output</span> <span class="o">=</span> <span class="n">embedded</span>
        <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gru</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">hidden</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span>

    <span class="k">def</span> <span class="nf">initHidden</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="n">torch</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="the-decoder">
<h3>The Decoder<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-decoder" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h3>
<p>The decoder is another RNN that takes the encoder output vector(s) and
outputs a sequence of words to create the translation.</p>
<div class="section" id="simple-decoder">
<h4>Simple Decoder<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#simple-decoder" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h4>
<p>In the simplest seq2seq decoder we use only last output of the encoder.
This last output is sometimes called the <em>context vector</em> as it encodes
context from the entire sequence. This context vector is used as the
initial hidden state of the decoder.</p>
<p>At every step of decoding, the decoder is given an input token and
hidden state. The initial input token is the start-of-string <code class="docutils literal notranslate"><span class="pre">&lt;SOS&gt;</span></code>
token, and the first hidden state is the context vector (the encoder’s
last hidden state).</p>
<div class="figure">
<img alt="" src="./seq2seq_files/decoder-network.png">
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DecoderRNN</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">,</span> <span class="n">output_size</span><span class="p">):</span>
        <span class="nb">super</span><span class="p">(</span><span class="n">DecoderRNN</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span> <span class="o">=</span> <span class="n">hidden_size</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">embedding</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="n">output_size</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">gru</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">GRU</span><span class="p">(</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">out</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">output_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">softmax</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">LogSoftmax</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">input</span><span class="p">,</span> <span class="n">hidden</span><span class="p">):</span>
        <span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">embedding</span><span class="p">(</span><span class="nb">input</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
        <span class="n">output</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">output</span><span class="p">)</span>
        <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gru</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">hidden</span><span class="p">)</span>
        <span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">output</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span>
        <span class="k">return</span> <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span>

    <span class="k">def</span> <span class="nf">initHidden</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="n">torch</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
</pre></div>
</div>
<p>I encourage you to train and observe the results of this model, but to
save space we’ll be going straight for the gold and introducing the
Attention Mechanism.</p>
</div>
<div class="section" id="attention-decoder">
<h4>Attention Decoder<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h4>
<p>If only the context vector is passed betweeen the encoder and decoder,
that single vector carries the burden of encoding the entire sentence.</p>
<p>Attention allows the decoder network to “focus” on a different part of
the encoder’s outputs for every step of the decoder’s own outputs. First
we calculate a set of <em>attention weights</em>. These will be multiplied by
the encoder output vectors to create a weighted combination. The result
(called <code class="docutils literal notranslate"><span class="pre">attn_applied</span></code> in the code) should contain information about
that specific part of the input sequence, and thus help the decoder
choose the right output words.</p>
<div class="figure">
<img alt="" src="./seq2seq_files/1152PYf.png">
</div>
<p>Calculating the attention weights is done with another feed-forward
layer <code class="docutils literal notranslate"><span class="pre">attn</span></code>, using the decoder’s input and hidden state as inputs.
Because there are sentences of all sizes in the training data, to
actually create and train this layer we have to choose a maximum
sentence length (input length, for encoder outputs) that it can apply
to. Sentences of the maximum length will use all the attention weights,
while shorter sentences will only use the first few.</p>
<div class="figure">
<img alt="" src="./seq2seq_files/attention-decoder-network.png">
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">AttnDecoderRNN</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">,</span> <span class="n">output_size</span><span class="p">,</span> <span class="n">dropout_p</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="n">MAX_LENGTH</span><span class="p">):</span>
        <span class="nb">super</span><span class="p">(</span><span class="n">AttnDecoderRNN</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span> <span class="o">=</span> <span class="n">hidden_size</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_size</span> <span class="o">=</span> <span class="n">output_size</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">dropout_p</span> <span class="o">=</span> <span class="n">dropout_p</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">max_length</span> <span class="o">=</span> <span class="n">max_length</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">embedding</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">attn</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span> <span class="o">*</span> <span class="mi">2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_length</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">attn_combine</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span> <span class="o">*</span> <span class="mi">2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dropout_p</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">gru</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">GRU</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">out</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_size</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">input</span><span class="p">,</span> <span class="n">hidden</span><span class="p">,</span> <span class="n">encoder_outputs</span><span class="p">):</span>
        <span class="n">embedded</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">embedding</span><span class="p">(</span><span class="nb">input</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
        <span class="n">embedded</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span><span class="p">(</span><span class="n">embedded</span><span class="p">)</span>

        <span class="n">attn_weights</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">attn</span><span class="p">(</span><span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">((</span><span class="n">embedded</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">hidden</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="mi">1</span><span class="p">)),</span> <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
        <span class="n">attn_applied</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">bmm</span><span class="p">(</span><span class="n">attn_weights</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">),</span>
                                 <span class="n">encoder_outputs</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span>

        <span class="n">output</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">((</span><span class="n">embedded</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">attn_applied</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="mi">1</span><span class="p">)</span>
        <span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">attn_combine</span><span class="p">(</span><span class="n">output</span><span class="p">)</span><span class="o">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>

        <span class="n">output</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">output</span><span class="p">)</span>
        <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gru</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">hidden</span><span class="p">)</span>

        <span class="n">output</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">log_softmax</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">output</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span><span class="p">,</span> <span class="n">attn_weights</span>

    <span class="k">def</span> <span class="nf">initHidden</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="n">torch</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">There are other forms of attention that work around the length
limitation by using a relative position approach. Read about “local
attention” in <a class="reference external" href="https://arxiv.org/abs/1508.04025">Effective Approaches to Attention-based Neural Machine
Translation</a>.</p>
</div>
</div>
</div>
</div>
<div class="section" id="training">
<h2>Training<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h2>
<div class="section" id="preparing-training-data">
<h3>Preparing Training Data<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#preparing-training-data" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h3>
<p>To train, for each pair we will need an input tensor (indexes of the
words in the input sentence) and target tensor (indexes of the words in
the target sentence). While creating these vectors we will append the
EOS token to both sequences.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">indexesFromSentence</span><span class="p">(</span><span class="n">lang</span><span class="p">,</span> <span class="n">sentence</span><span class="p">):</span>
    <span class="k">return</span> <span class="p">[</span><span class="n">lang</span><span class="o">.</span><span class="n">word2index</span><span class="p">[</span><span class="n">word</span><span class="p">]</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">sentence</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">)]</span>


<span class="k">def</span> <span class="nf">tensorFromSentence</span><span class="p">(</span><span class="n">lang</span><span class="p">,</span> <span class="n">sentence</span><span class="p">):</span>
    <span class="n">indexes</span> <span class="o">=</span> <span class="n">indexesFromSentence</span><span class="p">(</span><span class="n">lang</span><span class="p">,</span> <span class="n">sentence</span><span class="p">)</span>
    <span class="n">indexes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">EOS_token</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">(</span><span class="n">indexes</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">torch</span><span class="o">.</span><span class="n">long</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>


<span class="k">def</span> <span class="nf">tensorsFromPair</span><span class="p">(</span><span class="n">pair</span><span class="p">):</span>
    <span class="n">input_tensor</span> <span class="o">=</span> <span class="n">tensorFromSentence</span><span class="p">(</span><span class="n">input_lang</span><span class="p">,</span> <span class="n">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
    <span class="n">target_tensor</span> <span class="o">=</span> <span class="n">tensorFromSentence</span><span class="p">(</span><span class="n">output_lang</span><span class="p">,</span> <span class="n">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
    <span class="k">return</span> <span class="p">(</span><span class="n">input_tensor</span><span class="p">,</span> <span class="n">target_tensor</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="training-the-model">
<h3>Training the Model<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-the-model" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h3>
<p>To train we run the input sentence through the encoder, and keep track
of every output and the latest hidden state. Then the decoder is given
the <code class="docutils literal notranslate"><span class="pre">&lt;SOS&gt;</span></code> token as its first input, and the last hidden state of the
encoder as its first hidden state.</p>
<p>“Teacher forcing” is the concept of using the real target outputs as
each next input, instead of using the decoder’s guess as the next input.
Using teacher forcing causes it to converge faster but <a class="reference external" href="http://minds.jacobs-university.de/sites/default/files/uploads/papers/ESNTutorialRev.pdf">when the trained
network is exploited, it may exhibit
instability</a>.</p>
<p>You can observe outputs of teacher-forced networks that read with
coherent grammar but wander far from the correct translation -
intuitively it has learned to represent the output grammar and can “pick
up” the meaning once the teacher tells it the first few words, but it
has not properly learned how to create the sentence from the translation
in the first place.</p>
<p>Because of the freedom PyTorch’s autograd gives us, we can randomly
choose to use teacher forcing or not with a simple if statement. Turn
<code class="docutils literal notranslate"><span class="pre">teacher_forcing_ratio</span></code> up to use more of it.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">teacher_forcing_ratio</span> <span class="o">=</span> <span class="mf">0.5</span>


<span class="k">def</span> <span class="nf">train</span><span class="p">(</span><span class="n">input_tensor</span><span class="p">,</span> <span class="n">target_tensor</span><span class="p">,</span> <span class="n">encoder</span><span class="p">,</span> <span class="n">decoder</span><span class="p">,</span> <span class="n">encoder_optimizer</span><span class="p">,</span> <span class="n">decoder_optimizer</span><span class="p">,</span> <span class="n">criterion</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="n">MAX_LENGTH</span><span class="p">):</span>
    <span class="n">encoder_hidden</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">initHidden</span><span class="p">()</span>

    <span class="n">encoder_optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span>
    <span class="n">decoder_optimizer</span><span class="o">.</span><span class="n">zero_grad</span><span class="p">()</span>

    <span class="n">input_length</span> <span class="o">=</span> <span class="n">input_tensor</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
    <span class="n">target_length</span> <span class="o">=</span> <span class="n">target_tensor</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>

    <span class="n">encoder_outputs</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">max_length</span><span class="p">,</span> <span class="n">encoder</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>

    <span class="n">loss</span> <span class="o">=</span> <span class="mi">0</span>

    <span class="k">for</span> <span class="n">ei</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">input_length</span><span class="p">):</span>
        <span class="n">encoder_output</span><span class="p">,</span> <span class="n">encoder_hidden</span> <span class="o">=</span> <span class="n">encoder</span><span class="p">(</span>
            <span class="n">input_tensor</span><span class="p">[</span><span class="n">ei</span><span class="p">],</span> <span class="n">encoder_hidden</span><span class="p">)</span>
        <span class="n">encoder_outputs</span><span class="p">[</span><span class="n">ei</span><span class="p">]</span> <span class="o">=</span> <span class="n">encoder_output</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>

    <span class="n">decoder_input</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">([[</span><span class="n">SOS_token</span><span class="p">]],</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>

    <span class="n">decoder_hidden</span> <span class="o">=</span> <span class="n">encoder_hidden</span>

    <span class="n">use_teacher_forcing</span> <span class="o">=</span> <span class="kc">True</span> <span class="k">if</span> <span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">()</span> <span class="o">&lt;</span> <span class="n">teacher_forcing_ratio</span> <span class="k">else</span> <span class="kc">False</span>

    <span class="k">if</span> <span class="n">use_teacher_forcing</span><span class="p">:</span>
        <span class="c1"># Teacher forcing: Feed the target as the next input</span>
        <span class="k">for</span> <span class="n">di</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">target_length</span><span class="p">):</span>
            <span class="n">decoder_output</span><span class="p">,</span> <span class="n">decoder_hidden</span><span class="p">,</span> <span class="n">decoder_attention</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">(</span>
                <span class="n">decoder_input</span><span class="p">,</span> <span class="n">decoder_hidden</span><span class="p">,</span> <span class="n">encoder_outputs</span><span class="p">)</span>
            <span class="n">loss</span> <span class="o">+=</span> <span class="n">criterion</span><span class="p">(</span><span class="n">decoder_output</span><span class="p">,</span> <span class="n">target_tensor</span><span class="p">[</span><span class="n">di</span><span class="p">])</span>
            <span class="n">decoder_input</span> <span class="o">=</span> <span class="n">target_tensor</span><span class="p">[</span><span class="n">di</span><span class="p">]</span>  <span class="c1"># Teacher forcing</span>

    <span class="k">else</span><span class="p">:</span>
        <span class="c1"># Without teacher forcing: use its own predictions as the next input</span>
        <span class="k">for</span> <span class="n">di</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">target_length</span><span class="p">):</span>
            <span class="n">decoder_output</span><span class="p">,</span> <span class="n">decoder_hidden</span><span class="p">,</span> <span class="n">decoder_attention</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">(</span>
                <span class="n">decoder_input</span><span class="p">,</span> <span class="n">decoder_hidden</span><span class="p">,</span> <span class="n">encoder_outputs</span><span class="p">)</span>
            <span class="n">topv</span><span class="p">,</span> <span class="n">topi</span> <span class="o">=</span> <span class="n">decoder_output</span><span class="o">.</span><span class="n">topk</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
            <span class="n">decoder_input</span> <span class="o">=</span> <span class="n">topi</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span>  <span class="c1"># detach from history as input</span>

            <span class="n">loss</span> <span class="o">+=</span> <span class="n">criterion</span><span class="p">(</span><span class="n">decoder_output</span><span class="p">,</span> <span class="n">target_tensor</span><span class="p">[</span><span class="n">di</span><span class="p">])</span>
            <span class="k">if</span> <span class="n">decoder_input</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">==</span> <span class="n">EOS_token</span><span class="p">:</span>
                <span class="k">break</span>

    <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span>

    <span class="n">encoder_optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span>
    <span class="n">decoder_optimizer</span><span class="o">.</span><span class="n">step</span><span class="p">()</span>

    <span class="k">return</span> <span class="n">loss</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">/</span> <span class="n">target_length</span>
</pre></div>
</div>
<p>This is a helper function to print time elapsed and estimated time
remaining given the current time and progress %.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">math</span>


<span class="k">def</span> <span class="nf">asMinutes</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
    <span class="n">m</span> <span class="o">=</span> <span class="n">math</span><span class="o">.</span><span class="n">floor</span><span class="p">(</span><span class="n">s</span> <span class="o">/</span> <span class="mi">60</span><span class="p">)</span>
    <span class="n">s</span> <span class="o">-=</span> <span class="n">m</span> <span class="o">*</span> <span class="mi">60</span>
    <span class="k">return</span> <span class="s1">'</span><span class="si">%d</span><span class="s1">m </span><span class="si">%d</span><span class="s1">s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>


<span class="k">def</span> <span class="nf">timeSince</span><span class="p">(</span><span class="n">since</span><span class="p">,</span> <span class="n">percent</span><span class="p">):</span>
    <span class="n">now</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">now</span> <span class="o">-</span> <span class="n">since</span>
    <span class="n">es</span> <span class="o">=</span> <span class="n">s</span> <span class="o">/</span> <span class="p">(</span><span class="n">percent</span><span class="p">)</span>
    <span class="n">rs</span> <span class="o">=</span> <span class="n">es</span> <span class="o">-</span> <span class="n">s</span>
    <span class="k">return</span> <span class="s1">'</span><span class="si">%s</span><span class="s1"> (- </span><span class="si">%s</span><span class="s1">)'</span> <span class="o">%</span> <span class="p">(</span><span class="n">asMinutes</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="n">asMinutes</span><span class="p">(</span><span class="n">rs</span><span class="p">))</span>
</pre></div>
</div>
<p>The whole training process looks like this:</p>
<ul class="simple">
<li>Start a timer</li>
<li>Initialize optimizers and criterion</li>
<li>Create set of training pairs</li>
<li>Start empty losses array for plotting</li>
</ul>
<p>Then we call <code class="docutils literal notranslate"><span class="pre">train</span></code> many times and occasionally print the progress (%
of examples, time so far, estimated time) and average loss.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">trainIters</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">decoder</span><span class="p">,</span> <span class="n">n_iters</span><span class="p">,</span> <span class="n">print_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">plot_every</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">):</span>
    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
    <span class="n">plot_losses</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">print_loss_total</span> <span class="o">=</span> <span class="mi">0</span>  <span class="c1"># Reset every print_every</span>
    <span class="n">plot_loss_total</span> <span class="o">=</span> <span class="mi">0</span>  <span class="c1"># Reset every plot_every</span>

    <span class="n">encoder_optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">encoder</span><span class="o">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">)</span>
    <span class="n">decoder_optimizer</span> <span class="o">=</span> <span class="n">optim</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">decoder</span><span class="o">.</span><span class="n">parameters</span><span class="p">(),</span> <span class="n">lr</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">)</span>
    <span class="n">training_pairs</span> <span class="o">=</span> <span class="p">[</span><span class="n">tensorsFromPair</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">pairs</span><span class="p">))</span>
                      <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_iters</span><span class="p">)]</span>
    <span class="n">criterion</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">NLLLoss</span><span class="p">()</span>

    <span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">n_iters</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>
        <span class="n">training_pair</span> <span class="o">=</span> <span class="n">training_pairs</span><span class="p">[</span><span class="nb">iter</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span>
        <span class="n">input_tensor</span> <span class="o">=</span> <span class="n">training_pair</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
        <span class="n">target_tensor</span> <span class="o">=</span> <span class="n">training_pair</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>

        <span class="n">loss</span> <span class="o">=</span> <span class="n">train</span><span class="p">(</span><span class="n">input_tensor</span><span class="p">,</span> <span class="n">target_tensor</span><span class="p">,</span> <span class="n">encoder</span><span class="p">,</span>
                     <span class="n">decoder</span><span class="p">,</span> <span class="n">encoder_optimizer</span><span class="p">,</span> <span class="n">decoder_optimizer</span><span class="p">,</span> <span class="n">criterion</span><span class="p">)</span>
        <span class="n">print_loss_total</span> <span class="o">+=</span> <span class="n">loss</span>
        <span class="n">plot_loss_total</span> <span class="o">+=</span> <span class="n">loss</span>

        <span class="k">if</span> <span class="nb">iter</span> <span class="o">%</span> <span class="n">print_every</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
            <span class="n">print_loss_avg</span> <span class="o">=</span> <span class="n">print_loss_total</span> <span class="o">/</span> <span class="n">print_every</span>
            <span class="n">print_loss_total</span> <span class="o">=</span> <span class="mi">0</span>
            <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1"> (</span><span class="si">%d</span><span class="s1"> </span><span class="si">%d%%</span><span class="s1">) </span><span class="si">%.4f</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">timeSince</span><span class="p">(</span><span class="n">start</span><span class="p">,</span> <span class="nb">iter</span> <span class="o">/</span> <span class="n">n_iters</span><span class="p">),</span>
                                         <span class="nb">iter</span><span class="p">,</span> <span class="nb">iter</span> <span class="o">/</span> <span class="n">n_iters</span> <span class="o">*</span> <span class="mi">100</span><span class="p">,</span> <span class="n">print_loss_avg</span><span class="p">))</span>

        <span class="k">if</span> <span class="nb">iter</span> <span class="o">%</span> <span class="n">plot_every</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
            <span class="n">plot_loss_avg</span> <span class="o">=</span> <span class="n">plot_loss_total</span> <span class="o">/</span> <span class="n">plot_every</span>
            <span class="n">plot_losses</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">plot_loss_avg</span><span class="p">)</span>
            <span class="n">plot_loss_total</span> <span class="o">=</span> <span class="mi">0</span>

    <span class="n">showPlot</span><span class="p">(</span><span class="n">plot_losses</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="plotting-results">
<h3>Plotting results<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#plotting-results" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h3>
<p>Plotting is done with matplotlib, using the array of loss values
<code class="docutils literal notranslate"><span class="pre">plot_losses</span></code> saved while training.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="n">plt</span><span class="o">.</span><span class="n">switch_backend</span><span class="p">(</span><span class="s1">'agg'</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">matplotlib.ticker</span> <span class="k">as</span> <span class="nn">ticker</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>


<span class="k">def</span> <span class="nf">showPlot</span><span class="p">(</span><span class="n">points</span><span class="p">):</span>
    <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
    <span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
    <span class="c1"># this locator puts ticks at regular intervals</span>
    <span class="n">loc</span> <span class="o">=</span> <span class="n">ticker</span><span class="o">.</span><span class="n">MultipleLocator</span><span class="p">(</span><span class="n">base</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
    <span class="n">ax</span><span class="o">.</span><span class="n">yaxis</span><span class="o">.</span><span class="n">set_major_locator</span><span class="p">(</span><span class="n">loc</span><span class="p">)</span>
    <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">points</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="evaluation">
<h2>Evaluation<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#evaluation" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h2>
<p>Evaluation is mostly the same as training, but there are no targets so
we simply feed the decoder’s predictions back to itself for each step.
Every time it predicts a word we add it to the output string, and if it
predicts the EOS token we stop there. We also store the decoder’s
attention outputs for display later.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">evaluate</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">decoder</span><span class="p">,</span> <span class="n">sentence</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="n">MAX_LENGTH</span><span class="p">):</span>
    <span class="k">with</span> <span class="n">torch</span><span class="o">.</span><span class="n">no_grad</span><span class="p">():</span>
        <span class="n">input_tensor</span> <span class="o">=</span> <span class="n">tensorFromSentence</span><span class="p">(</span><span class="n">input_lang</span><span class="p">,</span> <span class="n">sentence</span><span class="p">)</span>
        <span class="n">input_length</span> <span class="o">=</span> <span class="n">input_tensor</span><span class="o">.</span><span class="n">size</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
        <span class="n">encoder_hidden</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">initHidden</span><span class="p">()</span>

        <span class="n">encoder_outputs</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">max_length</span><span class="p">,</span> <span class="n">encoder</span><span class="o">.</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>

        <span class="k">for</span> <span class="n">ei</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">input_length</span><span class="p">):</span>
            <span class="n">encoder_output</span><span class="p">,</span> <span class="n">encoder_hidden</span> <span class="o">=</span> <span class="n">encoder</span><span class="p">(</span><span class="n">input_tensor</span><span class="p">[</span><span class="n">ei</span><span class="p">],</span>
                                                     <span class="n">encoder_hidden</span><span class="p">)</span>
            <span class="n">encoder_outputs</span><span class="p">[</span><span class="n">ei</span><span class="p">]</span> <span class="o">+=</span> <span class="n">encoder_output</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>

        <span class="n">decoder_input</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">tensor</span><span class="p">([[</span><span class="n">SOS_token</span><span class="p">]],</span> <span class="n">device</span><span class="o">=</span><span class="n">device</span><span class="p">)</span>  <span class="c1"># SOS</span>

        <span class="n">decoder_hidden</span> <span class="o">=</span> <span class="n">encoder_hidden</span>

        <span class="n">decoded_words</span> <span class="o">=</span> <span class="p">[]</span>
        <span class="n">decoder_attentions</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">max_length</span><span class="p">,</span> <span class="n">max_length</span><span class="p">)</span>

        <span class="k">for</span> <span class="n">di</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">max_length</span><span class="p">):</span>
            <span class="n">decoder_output</span><span class="p">,</span> <span class="n">decoder_hidden</span><span class="p">,</span> <span class="n">decoder_attention</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">(</span>
                <span class="n">decoder_input</span><span class="p">,</span> <span class="n">decoder_hidden</span><span class="p">,</span> <span class="n">encoder_outputs</span><span class="p">)</span>
            <span class="n">decoder_attentions</span><span class="p">[</span><span class="n">di</span><span class="p">]</span> <span class="o">=</span> <span class="n">decoder_attention</span><span class="o">.</span><span class="n">data</span>
            <span class="n">topv</span><span class="p">,</span> <span class="n">topi</span> <span class="o">=</span> <span class="n">decoder_output</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">topk</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">topi</span><span class="o">.</span><span class="n">item</span><span class="p">()</span> <span class="o">==</span> <span class="n">EOS_token</span><span class="p">:</span>
                <span class="n">decoded_words</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'&lt;EOS&gt;'</span><span class="p">)</span>
                <span class="k">break</span>
            <span class="k">else</span><span class="p">:</span>
                <span class="n">decoded_words</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">output_lang</span><span class="o">.</span><span class="n">index2word</span><span class="p">[</span><span class="n">topi</span><span class="o">.</span><span class="n">item</span><span class="p">()])</span>

            <span class="n">decoder_input</span> <span class="o">=</span> <span class="n">topi</span><span class="o">.</span><span class="n">squeeze</span><span class="p">()</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span>

        <span class="k">return</span> <span class="n">decoded_words</span><span class="p">,</span> <span class="n">decoder_attentions</span><span class="p">[:</span><span class="n">di</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span>
</pre></div>
</div>
<p>We can evaluate random sentences from the training set and print out the
input, target, and output to make some subjective quality judgements:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">evaluateRandomly</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">decoder</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="mi">10</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
        <span class="n">pair</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">pairs</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">'&gt;'</span><span class="p">,</span> <span class="n">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">'='</span><span class="p">,</span> <span class="n">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
        <span class="n">output_words</span><span class="p">,</span> <span class="n">attentions</span> <span class="o">=</span> <span class="n">evaluate</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">decoder</span><span class="p">,</span> <span class="n">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
        <span class="n">output_sentence</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">output_words</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">'&lt;'</span><span class="p">,</span> <span class="n">output_sentence</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">''</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="training-and-evaluating">
<h2>Training and Evaluating<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-and-evaluating" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h2>
<p>With all these helper functions in place (it looks like extra work, but
it makes it easier to run multiple experiments) we can actually
initialize a network and start training.</p>
<p>Remember that the input sentences were heavily filtered. For this small
dataset we can use relatively small networks of 256 hidden nodes and a
single GRU layer. After about 40 minutes on a MacBook CPU we’ll get some
reasonable results.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If you run this notebook you can train, interrupt the kernel,
evaluate, and continue training later. Comment out the lines where the
encoder and decoder are initialized and run <code class="docutils literal notranslate"><span class="pre">trainIters</span></code> again.</p>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">hidden_size</span> <span class="o">=</span> <span class="mi">256</span>
<span class="n">encoder1</span> <span class="o">=</span> <span class="n">EncoderRNN</span><span class="p">(</span><span class="n">input_lang</span><span class="o">.</span><span class="n">n_words</span><span class="p">,</span> <span class="n">hidden_size</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>
<span class="n">attn_decoder1</span> <span class="o">=</span> <span class="n">AttnDecoderRNN</span><span class="p">(</span><span class="n">hidden_size</span><span class="p">,</span> <span class="n">output_lang</span><span class="o">.</span><span class="n">n_words</span><span class="p">,</span> <span class="n">dropout_p</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span>

<span class="n">trainIters</span><span class="p">(</span><span class="n">encoder1</span><span class="p">,</span> <span class="n">attn_decoder1</span><span class="p">,</span> <span class="mi">75000</span><span class="p">,</span> <span class="n">print_every</span><span class="o">=</span><span class="mi">5000</span><span class="p">)</span>
</pre></div>
</div>
<ul class="sphx-glr-horizontal">
<li><img alt="../_images/sphx_glr_seq2seq_translation_tutorial_001.png" class="sphx-glr-multi-img first" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_001.png">
</li>
<li><img alt="../_images/sphx_glr_seq2seq_translation_tutorial_002.png" class="sphx-glr-multi-img first" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_002.png">
</li>
</ul>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>1m 50s (- 25m 44s) (5000 6%) 2.8395
3m 35s (- 23m 21s) (10000 13%) 2.2645
5m 22s (- 21m 29s) (15000 20%) 1.9822
7m 10s (- 19m 42s) (20000 26%) 1.7177
8m 56s (- 17m 53s) (25000 33%) 1.5150
10m 44s (- 16m 7s) (30000 40%) 1.3715
12m 31s (- 14m 18s) (35000 46%) 1.2330
14m 16s (- 12m 29s) (40000 53%) 1.0873
16m 3s (- 10m 42s) (45000 60%) 0.9720
17m 49s (- 8m 54s) (50000 66%) 0.8693
19m 35s (- 7m 7s) (55000 73%) 0.8073
21m 21s (- 5m 20s) (60000 80%) 0.7157
23m 8s (- 3m 33s) (65000 86%) 0.6667
24m 56s (- 1m 46s) (70000 93%) 0.6293
26m 41s (- 0m 0s) (75000 100%) 0.5715
</pre></div>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">evaluateRandomly</span><span class="p">(</span><span class="n">encoder1</span><span class="p">,</span> <span class="n">attn_decoder1</span><span class="p">)</span>
</pre></div>
</div>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&gt; je suis optimiste de nature .
= i am an optimist by nature .
&lt; i am an optimist by nature . &lt;EOS&gt;

&gt; nous sommes decues .
= we re disappointed .
&lt; we re disappointed . &lt;EOS&gt;

&gt; je suis heureux que tu aimes mes amis .
= i m glad you liked my friends .
&lt; i m glad you liked my friends . &lt;EOS&gt;

&gt; je suis totalement d accord .
= i m in absolutely total agreement .
&lt; i m in on his . . &lt;EOS&gt;

&gt; nous n allons nulle part .
= we re not going anywhere .
&lt; we re not going anywhere . &lt;EOS&gt;

&gt; vous etes fort talentueuses .
= you re very talented .
&lt; you re very talented . &lt;EOS&gt;

&gt; je suis trop fatigue pour courir .
= i am too tired to run .
&lt; i m too tired to talk . &lt;EOS&gt;

&gt; il est gentil .
= he s a kind person .
&lt; he s kind of cute . &lt;EOS&gt;

&gt; nous sommes toutes dingues .
= we re all crazy .
&lt; we re all crazy . &lt;EOS&gt;

&gt; il cherche a plaire .
= he is eager to please .
&lt; he is eager to please . &lt;EOS&gt;
</pre></div>
</div>
<div class="section" id="visualizing-attention">
<h3>Visualizing Attention<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#visualizing-attention" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h3>
<p>A useful property of the attention mechanism is its highly interpretable
outputs. Because it is used to weight specific encoder outputs of the
input sequence, we can imagine looking where the network is focused most
at each time step.</p>
<p>You could simply run <code class="docutils literal notranslate"><span class="pre">plt.matshow(attentions)</span></code> to see attention output
displayed as a matrix, with the columns being input steps and rows being
output steps:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">output_words</span><span class="p">,</span> <span class="n">attentions</span> <span class="o">=</span> <span class="n">evaluate</span><span class="p">(</span>
    <span class="n">encoder1</span><span class="p">,</span> <span class="n">attn_decoder1</span><span class="p">,</span> <span class="s2">"je suis trop froid ."</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">matshow</span><span class="p">(</span><span class="n">attentions</span><span class="o">.</span><span class="n">numpy</span><span class="p">())</span>
</pre></div>
</div>
<img alt="../_images/sphx_glr_seq2seq_translation_tutorial_003.png" class="sphx-glr-single-img" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_003.png">
<p>For a better viewing experience we will do the extra work of adding axes
and labels:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">showAttention</span><span class="p">(</span><span class="n">input_sentence</span><span class="p">,</span> <span class="n">output_words</span><span class="p">,</span> <span class="n">attentions</span><span class="p">):</span>
    <span class="c1"># Set up figure with colorbar</span>
    <span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
    <span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_subplot</span><span class="p">(</span><span class="mi">111</span><span class="p">)</span>
    <span class="n">cax</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">matshow</span><span class="p">(</span><span class="n">attentions</span><span class="o">.</span><span class="n">numpy</span><span class="p">(),</span> <span class="n">cmap</span><span class="o">=</span><span class="s1">'bone'</span><span class="p">)</span>
    <span class="n">fig</span><span class="o">.</span><span class="n">colorbar</span><span class="p">(</span><span class="n">cax</span><span class="p">)</span>

    <span class="c1"># Set up axes</span>
    <span class="n">ax</span><span class="o">.</span><span class="n">set_xticklabels</span><span class="p">([</span><span class="s1">''</span><span class="p">]</span> <span class="o">+</span> <span class="n">input_sentence</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span> <span class="o">+</span>
                       <span class="p">[</span><span class="s1">'&lt;EOS&gt;'</span><span class="p">],</span> <span class="n">rotation</span><span class="o">=</span><span class="mi">90</span><span class="p">)</span>
    <span class="n">ax</span><span class="o">.</span><span class="n">set_yticklabels</span><span class="p">([</span><span class="s1">''</span><span class="p">]</span> <span class="o">+</span> <span class="n">output_words</span><span class="p">)</span>

    <span class="c1"># Show label at every tick</span>
    <span class="n">ax</span><span class="o">.</span><span class="n">xaxis</span><span class="o">.</span><span class="n">set_major_locator</span><span class="p">(</span><span class="n">ticker</span><span class="o">.</span><span class="n">MultipleLocator</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
    <span class="n">ax</span><span class="o">.</span><span class="n">yaxis</span><span class="o">.</span><span class="n">set_major_locator</span><span class="p">(</span><span class="n">ticker</span><span class="o">.</span><span class="n">MultipleLocator</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>

    <span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>


<span class="k">def</span> <span class="nf">evaluateAndShowAttention</span><span class="p">(</span><span class="n">input_sentence</span><span class="p">):</span>
    <span class="n">output_words</span><span class="p">,</span> <span class="n">attentions</span> <span class="o">=</span> <span class="n">evaluate</span><span class="p">(</span>
        <span class="n">encoder1</span><span class="p">,</span> <span class="n">attn_decoder1</span><span class="p">,</span> <span class="n">input_sentence</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'input ='</span><span class="p">,</span> <span class="n">input_sentence</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'output ='</span><span class="p">,</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">output_words</span><span class="p">))</span>
    <span class="n">showAttention</span><span class="p">(</span><span class="n">input_sentence</span><span class="p">,</span> <span class="n">output_words</span><span class="p">,</span> <span class="n">attentions</span><span class="p">)</span>


<span class="n">evaluateAndShowAttention</span><span class="p">(</span><span class="s2">"elle a cinq ans de moins que moi ."</span><span class="p">)</span>

<span class="n">evaluateAndShowAttention</span><span class="p">(</span><span class="s2">"elle est trop petit ."</span><span class="p">)</span>

<span class="n">evaluateAndShowAttention</span><span class="p">(</span><span class="s2">"je ne crains pas de mourir ."</span><span class="p">)</span>

<span class="n">evaluateAndShowAttention</span><span class="p">(</span><span class="s2">"c est un jeune directeur plein de talent ."</span><span class="p">)</span>
</pre></div>
</div>
<ul class="sphx-glr-horizontal">
<li><img alt="../_images/sphx_glr_seq2seq_translation_tutorial_004.png" class="sphx-glr-multi-img first" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_004.png">
</li>
<li><img alt="../_images/sphx_glr_seq2seq_translation_tutorial_005.png" class="sphx-glr-multi-img first" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_005.png">
</li>
<li><img alt="../_images/sphx_glr_seq2seq_translation_tutorial_006.png" class="sphx-glr-multi-img first" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_006.png">
</li>
<li><img alt="../_images/sphx_glr_seq2seq_translation_tutorial_007.png" class="sphx-glr-multi-img first" src="./seq2seq_files/sphx_glr_seq2seq_translation_tutorial_007.png">
</li>
</ul>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>input = elle a cinq ans de moins que moi .
output = she is five years younger than me . &lt;EOS&gt;
input = elle est trop petit .
output = she is too drunk . &lt;EOS&gt;
input = je ne crains pas de mourir .
output = i m not scared of dying . &lt;EOS&gt;
input = c est un jeune directeur plein de talent .
output = he s a talented young player . &lt;EOS&gt;
</pre></div>
</div>
</div>
</div>
<div class="section" id="exercises">
<h2>Exercises<a class="anchorjs-link " href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#exercises" aria-label="Anchor" data-anchorjs-icon="" style="font: 1em/1 anchorjs-icons; padding-left: 0.375em;"></a></h2>
<ul class="simple">
<li>Try with a different dataset<ul>
<li>Another language pair</li>
<li>Human → Machine (e.g. IOT commands)</li>
<li>Chat → Response</li>
<li>Question → Answer</li>
</ul>
</li>
<li>Replace the embeddings with pre-trained word embeddings such as word2vec or
GloVe</li>
<li>Try with more layers, more hidden units, and more sentences. Compare
the training time and results.</li>
<li>If you use a translation file where pairs have two of the same phrase
(<code class="docutils literal notranslate"><span class="pre">I</span> <span class="pre">am</span> <span class="pre">test</span> <span class="pre">\t</span> <span class="pre">I</span> <span class="pre">am</span> <span class="pre">test</span></code>), you can use this as an autoencoder. Try
this:<ul>
<li>Train as an autoencoder</li>
<li>Save only the Encoder network</li>
<li>Train a new Decoder for translation from there</li>
</ul>
</li>
</ul>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 26 minutes  55.048 seconds)</p>
<div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-intermediate-seq2seq-translation-tutorial-py">
<div class="sphx-glr-download docutils container">
<a class="reference download internal has-code" download="" href="https://pytorch.org/tutorials/_downloads/a96a2daac1918ec72f68233dfe3f2c47/seq2seq_translation_tutorial.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">seq2seq_translation_tutorial.py</span></code></a></div>
<div class="sphx-glr-download docutils container">
<a class="reference download internal has-code" download="" href="https://pytorch.org/tutorials/_downloads/a60617788061539b5449701ae76aee56/seq2seq_translation_tutorial.ipynb"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Jupyter</span> <span class="pre">notebook:</span> <span class="pre">seq2seq_translation_tutorial.ipynb</span></code></a></div>
</div>
<p class="sphx-glr-signature"><a class="reference external" href="https://sphinx-gallery.readthedocs.io/">Gallery generated by Sphinx-Gallery</a></p>
</div>
</div>
</article>
</div>
<footer>
<div aria-label="footer navigation" class="rst-footer-buttons" role="navigation">
<a accesskey="n" class="btn btn-neutral float-right" href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="next" title="DCGAN Tutorial">Next <img class="next-page" src="./seq2seq_files/chevron-right-orange.svg"></a>
<a accesskey="p" class="btn btn-neutral" href="https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html" rel="prev" title="Advanced: Making Dynamic Decisions and the Bi-LSTM CRF"><img class="previous-page" src="./seq2seq_files/chevron-right-orange.svg"> Previous</a>
</div>
<hr class="helpful-hr hr-top">
<div class="helpful-container">
<div class="helpful-question">Was this helpful?</div>
<div class="helpful-question yes-link" data-behavior="was-this-helpful-event" data-response="yes">Yes</div>
<div class="helpful-question no-link" data-behavior="was-this-helpful-event" data-response="no">No</div>
<div class="was-helpful-thank-you">Thank you</div>
</div>
<hr class="helpful-hr hr-bottom">
<div role="contentinfo">
<p>
        © Copyright 2017, PyTorch.

    </p>
</div>
<div>
        Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org/">Read the Docs</a>.
      </div>
</footer>
</div>
</div>
<div class="pytorch-content-right" id="pytorch-content-right" style="height: 100%;">
<div class="pytorch-right-menu" id="pytorch-right-menu" style="top: 0px;">
<div class="pytorch-side-scroll" id="pytorch-side-scroll-right" style="height: 465px;">
<ul>
<li><a class="reference internal title-link has-children" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#">Translation with a Sequence to Sequence Network and Attention</a><ul>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#loading-data-files">Loading data files</a></li>
<li><a class="reference internal not-expanded" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-seq2seq-model">The Seq2Seq Model</a><ul>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-encoder">The Encoder</a></li>
<li><a class="reference internal not-expanded" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#the-decoder">The Decoder</a><ul>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#simple-decoder">Simple Decoder</a></li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder">Attention Decoder</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal not-expanded" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training">Training</a><ul>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#preparing-training-data">Preparing Training Data</a></li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-the-model">Training the Model</a></li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#plotting-results">Plotting results</a></li>
</ul>
</li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#evaluation">Evaluation</a></li>
<li><a class="reference internal not-expanded" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-and-evaluating">Training and Evaluating</a><ul>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#visualizing-attention">Visualizing Attention</a></li>
</ul>
</li>
<li><a class="reference internal" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#exercises">Exercises</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div>
</section>
</div>
<script data-url_root="../" id="documentation_options" src="./seq2seq_files/documentation_options.js" type="text/javascript"></script>
<script src="./seq2seq_files/jquery.js" type="text/javascript"></script>
<script src="./seq2seq_files/underscore.js" type="text/javascript"></script>
<script src="./seq2seq_files/doctools.js" type="text/javascript"></script>
<script async="async" src="./seq2seq_files/MathJax.js" type="text/javascript"></script>
<script src="./seq2seq_files/popper.min.js" type="text/javascript"></script>
<script src="./seq2seq_files/bootstrap.min.js" type="text/javascript"></script>
<script src="./seq2seq_files/theme.js" type="text/javascript"></script>
<script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>
<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-90545585-2', 'auto');
  ga('send', 'pageview');

</script>
<script async="" src="./seq2seq_files/js"></script>
<script>

  window.dataLayer = window.dataLayer || [];

  function gtag(){dataLayer.push(arguments);}

  gtag('js', new Date());
  gtag('config', 'UA-117752657-2');

</script>
<script>
  !function(f,b,e,v,n,t,s)
  {if(f.fbq)return;n=f.fbq=function(){n.callMethod?
  n.callMethod.apply(n,arguments):n.queue.push(arguments)};
  if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0';
  n.queue=[];t=b.createElement(e);t.async=!0;
  t.src=v;s=b.getElementsByTagName(e)[0];
  s.parentNode.insertBefore(t,s)}(window,document,'script',
  'https://connect.facebook.net/en_US/fbevents.js');
  fbq('init', '243028289693773');
  fbq('track', 'PageView');

  $("[data-behavior='call-to-action-event']").on('click', function(){
    fbq('trackCustom', "Download", {
      tutorialTitle: $('h1:first').text(),
      downloadLink: this.href,
      tutorialLink: window.location.href,
      downloadTitle: $(this).attr("data-response")
    });
    ga('send', {
      hitType: 'event',
      eventCategory: 'Download',
      eventAction: 'click',
      eventLabel: $(this).attr("data-response")
    });
   });

   $("[data-behavior='was-this-helpful-event']").on('click', function(){
    $(".helpful-question").hide();
    $(".was-helpful-thank-you").show();
    fbq('trackCustom', "Was this Helpful?", {
      tutorialLink: window.location.href,
      tutorialTitle: $('h1:first').text(),
      helpful: $(this).attr("data-response")
    });
    ga('send', {
      hitType: 'event',
      eventCategory: 'Was this Helpful?',
      eventAction: 'click',
      eventLabel: $(this).attr("data-response")
    });
   });

   if (location.pathname == "/") {
     $(".helpful-container").hide();
     $(".hr-bottom").hide();
   }
</script>
<noscript>
<img height="1" src="https://www.facebook.com/tr?id=243028289693773&amp;ev=PageView
  &amp;noscript=1" width="1"/>
</noscript>
<img alt="" height="1" src="./seq2seq_files/saved_resource" style="border-style:none;" width="1">
<!-- Begin Footer -->
<div class="container-fluid docs-tutorials-resources" id="docs-tutorials-resources">
<div class="container">
<div class="row">
<div class="col-md-4 text-center">
<h2>Docs</h2>
<p>Access comprehensive developer documentation for PyTorch</p>
<a class="with-right-arrow" href="https://pytorch.org/docs/stable/index.html">View Docs</a>
</div>
<div class="col-md-4 text-center">
<h2>Tutorials</h2>
<p>Get in-depth tutorials for beginners and advanced developers</p>
<a class="with-right-arrow" href="https://pytorch.org/tutorials">View Tutorials</a>
</div>
<div class="col-md-4 text-center">
<h2>Resources</h2>
<p>Find development resources and get your questions answered</p>
<a class="with-right-arrow" href="https://pytorch.org/resources">View Resources</a>
</div>
</div>
</div>
</div>
<footer class="site-footer">
<div class="container footer-container">
<div class="footer-logo-wrapper">
<a class="footer-logo" href="https://pytorch.org/"></a>
</div>
<div class="footer-links-wrapper">
<div class="footer-links-col">
<ul>
<li class="list-title"><a href="https://pytorch.org/">PyTorch</a></li>
<li><a href="https://pytorch.org/get-started">Get Started</a></li>
<li><a href="https://pytorch.org/features">Features</a></li>
<li><a href="https://pytorch.org/ecosystem">Ecosystem</a></li>
<li><a href="https://pytorch.org/blog/">Blog</a></li>
<li><a href="https://pytorch.org/resources">Resources</a></li>
</ul>
</div>
<div class="footer-links-col">
<ul>
<li class="list-title"><a href="https://pytorch.org/support">Support</a></li>
<li><a href="https://pytorch.org/tutorials">Tutorials</a></li>
<li><a href="https://pytorch.org/docs/stable/index.html">Docs</a></li>
<li><a href="https://discuss.pytorch.org/" target="_blank">Discuss</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues" target="_blank">Github Issues</a></li>
<li><a href="https://pytorch.slack.com/" target="_blank">Slack</a></li>
<li><a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md" target="_blank">Contributing</a></li>
</ul>
</div>
<div class="footer-links-col follow-us-col">
<ul>
<li class="list-title">Follow Us</li>
<li>
<div id="mc_embed_signup">
<form action="https://twitter.us14.list-manage.com/subscribe/post?u=75419c71fe0a935e53dfa4a3f&amp;id=91d0dccd39" class="email-subscribe-form validate" id="mc-embedded-subscribe-form" method="post" name="mc-embedded-subscribe-form" novalidate="" target="_blank">
<div class="email-subscribe-form-fields-wrapper" id="mc_embed_signup_scroll">
<div class="mc-field-group">
<label for="mce-EMAIL" style="display:none;">Email Address</label>
<input class="required email" id="mce-EMAIL" name="EMAIL" placeholder="Email Address" type="email" value="">
</div>
<div class="clear" id="mce-responses">
<div class="response" id="mce-error-response" style="display:none"></div>
<div class="response" id="mce-success-response" style="display:none"></div>
</div> <!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div aria-hidden="true" style="position: absolute; left: -5000px;"><input name="b_75419c71fe0a935e53dfa4a3f_91d0dccd39" tabindex="-1" type="text" value=""></div>
<div class="clear">
<input class="button email-subscribe-button" id="mc-embedded-subscribe" name="subscribe" type="submit" value="">
</div>
</div>
</form>
</div>
</li>
</ul>
<div class="footer-social-icons">
<a class="facebook" href="https://www.facebook.com/pytorch" target="_blank"></a>
<a class="twitter" href="https://twitter.com/pytorch" target="_blank"></a>
</div>
</div>
</div>
</div>
</footer>
<div class="cookie-banner-wrapper">
<div class="container">
<p class="gdpr-notice">To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: <a href="https://www.facebook.com/policies/cookies/">Cookies Policy</a>.</p>
<img class="close-button" src="./seq2seq_files/pytorch-x.svg">
</div>
</div>
<!-- End Footer -->
<!-- Begin Mobile Menu -->
<div class="mobile-main-menu">
<div class="container-fluid">
<div class="container">
<div class="mobile-main-menu-header-container">
<a aria-label="PyTorch" class="header-logo" href="https://pytorch.org/"></a>
<a class="main-menu-close-button" data-behavior="close-mobile-menu" href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#"></a>
</div>
</div>
</div>
<div class="mobile-main-menu-links-container">
<div class="main-menu">
<ul>
<li>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#">Get Started</a>
</li>
<li>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#">Features</a>
</li>
<li>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#">Ecosystem</a>
</li>
<li>
<a href="https://pytorch.org/blog/">Blog</a>
</li>
<li class="active">
<a href="https://pytorch.org/tutorials">Tutorials</a>
</li>
<li>
<a href="https://pytorch.org/docs/stable/index.html">Docs</a>
</li>
<li>
<a href="https://pytorch.org/resources">Resources</a>
</li>
<li>
<a href="https://github.com/pytorch/pytorch">Github</a>
</li>
</ul>
</div>
</div>
</div>
<!-- End Mobile Menu -->
<script src="./seq2seq_files/anchor.min.js" type="text/javascript"></script>
<script type="text/javascript">
    $(document).ready(function() {
      mobileMenu.bind();
      mobileTOC.bind();
      pytorchAnchors.bind();
      sideMenus.bind();
      scrollToAnchor.bind();
      highlightNavigation.bind();

      // Add class to links that have code blocks, since we cannot create links in code blocks
      $("article.pytorch-article a span.pre").each(function(e) {
        $(this).closest("a").addClass("has-code");
      });
    })
  </script>

</body></html>